System and method for storing redundant information

ABSTRACT

A method and system for reducing storage requirements and speeding up storage operations by reducing the storage of redundant data includes receiving a request that identifies one or more data objects to which to apply a storage operation. For each data object, the storage system determines if the data object contains data that matches another data object to which the storage operation was previously applied. If the data objects do not match, then the storage system performs the storage operation in a usual manner. However, if the data objects do match, then the storage system may avoid performing the storage operation.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application of U.S. patentapplication Ser. No. 12/058,367 (Attorney Docket No. 60692.8036.US05),entitled “SYSTEM AND METHOD FOR STORING REDUNDANT INFORMATION,” filedMar. 28, 2008, which claims the benefit of U.S. Provisional PatentApplication No. 61/066,150 (Attorney Docket No. 60692-8036.US01),entitled “SYSTEM AND METHOD FOR STORING REDUNDANT INFORMATION,” filed onOct. 31, 2007, each of which is hereby incorporated by reference; andwherein U.S. patent application Ser. No. 12/058,367 is a continuationapplication of U.S. patent application Ser. No. 11/963,623 (AttorneyDocket No. 60692-8036.US02), entitled “SYSTEM AND METHOD FOR STORINGREDUNDANT INFORMATION,” filed on Dec. 21, 2007, which claims the benefitof U.S. Provisional Patent Application No. 60/871,737 (Attorney DocketNo. 60692-8036.US00), entitled “SYSTEM AND METHOD FOR STORING REDUNDANTINFORMATION,” filed on Dec. 22, 2006, each of which is herebyincorporated by reference.

BACKGROUND

Computer systems contain large amounts of information. This informationincludes personal information, such as financial information,customer/client/patient contact information, audio/visual information,and much more. This information also includes information related to thecorrect operation of the computer system, such as operating systemfiles, application files, user settings, and so on. With the increasedreliance on computer systems to store critical information, theimportance of protecting information has grown. Traditional storagesystems receive an identification of a file to protect, then create oneor more secondary copies, such as backup files, containing the contentsof the file. These secondary copies can then later be used to restorethe original data should anything happen to the original data.

In corporate environments, protecting information is generally part of aroutine process that is performed for many computer systems within anorganization. For example, a company might back up critical computingsystems related to e-commerce such as databases, file servers, webservers, and so on. The company may also protect computing systems usedby each of its employees, such as those used by an accountingdepartment, marketing department, engineering, and so forth.

Although each computer system may contain certain unique information,many systems may contain very similar information. For example, althougha computing system used by a marketing employee and another computingsystem used by an engineering employee will generally contain uniqueinformation created by each employee in the course of their work, bothcomputing systems will likely have the same operating system installed,with thousands of identical or similar files used by the operatingsystem. Similarly, both computing systems will likely have at least somesimilar application programs installed, such as a word processor,spreadsheet, Internet browser, and so on. Both systems may also havesimilar corporate information. For example, each employee may have anelectronic copy of an employee manual distributed by the company.Information other than files may also be identical or similar betweensystems. For example, user settings and preferences may have similardefault values on each system and application programs may containsimilar templates on each system that are stored as application-specificinformation. As another example, several employees may have received acopy of the same email, and the email may be stored in each employee'selectronic mailbox.

As a result of the amount of redundant information in an organization,secondary copies of an organization's information are often very largeand can require the purchase of expensive storage devices and storagemedia. The restoration of data in the event of data loss is also slowedby the large size of the secondary copies. As the size of secondarycopies increase, locating and restoring information requires moreactions to be taken to restore information. For example, it may benecessary to search many tapes or other media to find the correctsecondary copy. The great quantity of storage media, such as tapes, maymean that some secondary storage media has been moved offsite such thatit must first be retrieved before information can be recovered from it.Each of these factors increases the cost of protecting information andthe time required to recover information in the event of data loss.Quick recovery of information is often critical to today's businesses,and any additional delay could affect business operations and customers'satisfaction with the business.

There is a need for a system that overcomes the above problems, as wellas one that provides additional benefits.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram that illustrates components of a system in oneembodiment of the invention.

FIG. 2 is a block diagram that illustrates a flow of data during astorage operation, in one embodiment.

FIG. 3 is a block diagram that shows a typical arrangement of thestorage system in an enterprise, in one embodiment.

FIG. 4 is a flow diagram that illustrates processing of a storageoperation manager component of the system in one embodiment.

FIG. 5 is a flow diagram that illustrates processing of the system todetermine whether a file is unique, in one embodiment.

FIG. 6 is a flow diagram that illustrates processing of the storageoperation component to restore data, in one embodiment.

FIG. 7 illustrates the process of storing data from three clients firstto single-instanced random access media and then to sequential media.

FIG. 8A is a data structure that illustrates a layout of storage data ona tape or other sequential media, in one embodiment.

FIG. 8B is a data structure that illustrates a type of data stored ineach entry of FIG. 8A, in one embodiment.

FIG. 9 is a table that illustrates information stored by a singleinstance database component, in one embodiment.

FIG. 10 is a flow diagram that illustrates processing of the storageoperation component to restore date from sequential media, in oneembodiment.

In the drawings, the same reference numbers and acronyms identifyelements or acts with the same or similar functionality for ease ofunderstanding and convenience. To easily identify the discussion of anyparticular element or act, the most significant digit or digits in areference number refer to the Figure number in which that element isfirst introduced (e.g., element 604 is first introduced and discussedwith respect to FIG. 6).

DETAILED DESCRIPTION

The headings provided herein are for convenience only and do notnecessarily affect the scope or meaning of the claimed invention.

Overview

A method and system for reducing storage requirements and speeding upstorage operations by reducing storage of redundant data, referred to assingle instancing, first receives a request that identifies one or moredata objects on which to perform a storage operation. A storageoperation can include many different types of operations, such as abackup, copy, snapshot, disk image, and so on. For example, one storageoperation is a backup of data, and the request may specify the locationof data objects to be backed up, such as a directory of files, and alocation to store backup data, such as a network file server. A dataobject can include many different types of data, such as files, email,application-specific data (e.g., documents, spreadsheets, databases, andso on), configuration settings, and so forth. For each data object, thestorage system determines if the data object contains data that matchesanother data object that was the subject of a previous storageoperation. For example, if the storage operation is a backup of datafiles, then the storage system determines if backup data from a previousbackup operation already contains a particular file to be backed up bythe current operation. If the data objects do not match, then thestorage system performs the storage operation in a usual manner.However, if the data objects do match, then the storage system may avoidperforming the storage operation. For example, in the case of a backup,the storage system may only store the files that are not alreadycontained in backup data from a previous backup operation. Foradditional instances of the file, the storage system may store metadatadescribing the presence of the additional instances without storing theadditional instances themselves. For example, if the same file isdiscovered in two different directories as part of a storage operation,then the first instance of the file may be stored in total and aplaceholder or pointer may be stored representing the second instance ofthe file rather than storing the second instance of the file itself. Thedata objects may comprise any type of electronic information or objectthat supports object-level recovery, such as files, emails, attachments,and application data. Thus, the storage system reduces the amount ofstorage required to perform the storage operation and saves time byreducing the amount of data for which storage operations are performedto satisfy a storage request.

Single instancing presents a number of challenges when using sequentialmedia. Sequential media includes media such as tapes that are typicallyaccessed (e.g., read and written) sequentially. Sequential media iscontrasted with random-access media, such as disk drives, flash drives,and optical media, that can be accessed randomly. One difficulty whenusing sequential media is that seeking to a particular location takeslonger than random-access media. For example, while a hard drive mayhave an access time of several milliseconds, sequential media may havean access time of several seconds to several minutes. Therefore, it isoften desirable to lay out data on sequential media in a way thatreduces or eliminates dependencies among the data on different parts ofthe media. Single instancing, on the other hand, creates dependenciesbetween the original instance of a file and additional instances thatreference the original instance of the file. Thus, the storage systemprovides special techniques for storing and retrieving single instancedata on sequential media.

The invention will now be described with respect to various embodiments.The following description provides specific details for a thoroughunderstanding of, and enabling description for, these embodiments of theinvention. However, one skilled in the art will understand that theinvention may be practiced without these details. In other instances,well-known structures and functions have not been shown or described indetail to avoid unnecessarily obscuring the description of theembodiments of the invention.

The terminology used in the description presented below is intended tobe interpreted in its broadest reasonable manner, even though it isbeing used in conjunction with a detailed description of certainspecific embodiments of the invention. Certain terms may even beemphasized below; however, any terminology intended to be interpreted inany restricted manner will be overtly and specifically defined as suchin this Detailed Description section.

Determining Data Object Similarity

Various methods of determining if one data object is similar (e.g., twoinstances of the same data) to another data object will now bedescribed. However, those of ordinary skill in the art will recognizethat many other methods besides those described here can be used toachieve similar results.

In some embodiments, the storage system determines if two data objectsare similar by performing a binary comparison. For example, a first filecan be compared byte by byte with a second file. If each byte of data ineach file matches, then the two files are identical and thereforesimilar. Otherwise, the two files do not match and are not treated assimilar.

In some embodiments, the storage system determines if two data objectsare similar by creating a digest or fingerprint of the data contained ineach data object. For example, as storage operations are performed, thestorage system may perform a hash on each file to create a digest of thefile. The digest of the file can be compared with stored digests createdfor other files. If the digest of two files match, then the system mayconsider the files to be identical. Any suitable hashing algorithm canbe used, such as Message Digest version 5 (MD5).

In some embodiments, the storage system may store additional informationalong with a data object digest. A hash algorithm is generally selectedthat has a low likelihood of hash collisions, which occur when two dataobjects containing different data have the same hash value. However,hash collisions do occur and additional information can be used tosubstantially reduce the likelihood of these collisions. For example,for files the system may store the file size as additional information.When two data objects are compared to determine their similarity, thedigests are compared as well as the additional information. If both thedigests and the additional information match are the data objectsconsidered identical. For example, for two files if the hash value andsize do not match, then the files are not considered identical eventhough the hash values alone may match. Some information, such as thefile name for a file, may not be included in the digest so that thestorage system will consider two files having the same data butdifferent names identical. Even though the names of files or otherinformation may differ, the hash value will be the same and the storagesystem can avoid storing the redundant data. The storage system can alsoperform other operations after hashing is complete, such as encryptingthe data. By hashing the data before encryption or other operations thatmodify the data are performed, the storage system allows singleinstancing to be performed at some time later on the modified data. Forexample, if files are initially copied from a client system to a firstcopy, then hashed, and then encrypted to create a second copy, it isstill possible for the storage system to single instance the second copyand remove redundant instances of data based on the pre-generated hasheswithout having to first decrypt the data to restore its originalcontent.

In some embodiments, the storage system may operate specially on dataobjects that are similar, but not identical. For example, many computingsystems in an organization may have word processors with similardocument templates, but each user may have slightly modified thedocument templates to match that user's preferences. The storage systemmay compare the document templates of two users and determine whichportions are the same and which portions are different. Then, the systemmay treat the portions that are the same as described above, avoidingredundant storage of the identical portions, while storing only theportions that are different. As another example, several users may havereceived an email and replied to the email. The replies may contain somenew text, but much of the text may be the contents of the originalmessage. In this example, the storage system may only store the similartext once and store the new text for each user's reply. For example, thesystem may break the email up into blocks, and compare blocks within theemail message to determine which are similar.

Single Instancing—Auxiliary Copies

Reducing or eliminating redundant instances of data resulting from astorage operation is sometimes referred to here as “single instancing,”because what would traditionally be stored as many instances of the samedata is reduced to as few as one. Redundant instances may be detectedand reduced at several locations or times throughout the operation ofthe system that will now be described. These embodiments are providedonly as examples, and are not intended to be an exhaustive list of theway in which the system can be implemented.

In some embodiments, the storage system performs single instancing ofdata at the system containing or originally generating the data, such asa client system. For example, a client system that is the source of datathat is the subject of a storage operation may receive a request fromthe storage system to provide a hash value of each file to be includedin the storage operation. The client uses a hashing algorithm to producea hash of the files or other data objects requested by the storagesystem. Alternatively, the client may generate hash values itself foreach file that is stored on the client system on an ongoing or otherbasis. When a storage operation is requested, the storage systemreceives the hash values from the client and determines if it is alreadyaware of a file with a hash value matching that of one provided by theclient system. For example, the storage system may check an index ofhashes received from client systems and look for matches with the hashvalue or values received from the client. If the storage system isalready aware of a similar file or matching file, then the client systemdoes not even need to send the redundant file to the secondary storagelocation or destination. Thus, the storage system may send a follow uprequest to the client system that specifies a list of files that are newto the storage system. Then, the client sends the full content of thespecified files.

In some embodiments, the storage system performs single instancing ofdata at the subject system of a storage operation. In the example of astorage operation, the storage system may request that a client systemsend one or more files stored on the client system to a server system(e.g., a media agent). Then, the server system checks each file (e.g.,by comparing a hash of the data) received from the client system todetermine if an instance of the received file has already been stored onthe server system. Additional instances of a received file that thestorage system has already stored on the server system can be discarded,whereas new files can be stored for the first time and can be indexed onor tracked by the server system. This alternative method eliminates theresources used by the client system for computing digests or hashvalues, and does not require the client system to be modified from theconfiguration used for a traditional storage operation. However,additional network resources may be consumed by the extra copying, andthe work of computing digests or hash values is shifted to the serversystem.

In some embodiments, the storage system performs single instancing on acopy of data created by a previously performed storage operation. Forexample, the storage system may copy data from a primary location to asecondary location, and then perform single instancing on the secondarycopy of the data. The storage system may create many copies of data,sometimes referred to as secondary or auxiliary copies, as part of anorganization's data handling plan, and single instancing may beperformed at any stage of the process on any of the copies. The storagesystem may process the copied data, determine which files or other dataobjects are redundant with other data in the copy or with other copiesof the data, and then save a new copy of the data that does not containthe redundant information. In this way, a backup server or other systemneed not be modified to reduce storage requirements, but theorganization still benefits from single instancing by reducing thestorage requirements later in the process, such as before backup data isstored on tape or shipped offsite for storage. As another example, thesystem may be configured to receive backup data created by anothersystem and create a single instanced copy of the data that is smaller insize. For example, the system may be configured to process copies ofdata from popular data storage applications and reduce the size of thecopies of data. For example, the data from the other system may be in adifferent format. The system may contain agents capable of parsing thedata in its original format and processing it into a more usable format,and then single instancing the data. This allows an organization toreduce the storage requirements for data that may have been copied orbacked up a long time ago.

In some embodiments, the storage system or other system performsadditional operations on the data after single instancing has occurred.For example, backup data being stored offsite may be encrypted toprevent unauthorized parties from accessing the data. The data may alsobe compressed to further reduce its size. In this way, the storagesystem performs the additional operations more efficiently, becausethere is less data on which to perform the operation after redundantdata has been reduced or eliminated.

Storing Single Instance Data

Single instance data may be stored in a variety of different ways. Insome embodiments, single instance data may be stored in folders andfiles just like other, non-single instanced or file system data. When afirst folder contains a file that is already present in another folder,the first folder may contain a reference, pointer, or path to the file,or a stub file that contains information about where the file containingthe actual data can be located. For example, two folders A and B mayeach contain a file X. Folder A may contain the actual file X, whilefolder B may contain a stub of file X with file attributes or othersupplemental information that indicates that the stub is a reference tothe instance of file X located in folder A. The fact that the file infolder B is a stub may be transparent to an application. For example, afile system filter driver may detect the stub and provide the actualfile data from folder A to any application that attempts to access thefile in folder B. Thus, the storage system preserves the organization ofthe data, but eliminates the redundant instances of the data, thussaving storage space and time consumed performing additional storageoperations on the data.

In some embodiments, the single instance of the file may contain areference count or other count to track the number of other instancesthat are referring to the file. Alternatively, an index of files storedby the storage system may track the references to any particular singleinstanced file. In the example above, the file X in folder A may containa reference count of two indicating that the file belongs in twolocations, folder A folder B. The storage system may use the referencecount to prevent the file from being deleted or otherwise madeunavailable while there are still outstanding references to the file.

Single Instancing—Sequential Media

Tapes and other sequential or removable media are often used for backupand other storage operations. Sequential media present specialchallenges for storage systems because the drawbacks of accessing dataout of sequence are often much higher than accessing the data insequence. For example, with tape media it can be very fast to read datafrom the beginning of the tape, then the middle, then the end (i.e.,sequentially), but much slower to read data first from the end, then themiddle, then the beginning (i.e., randomly or out of sequence).Therefore, it is often a goal of storage systems to structure datastored to sequential media in such a way that the data can be accessedsequentially.

In some embodiments, the storage system prepares single instance copiesof data destined for sequential media using intermediate (e.g., randomaccess) media. For example, the storage system may perform a singleinstance backup to a hard drive, and then process the data stored on thehard drive to prepare it for writing to tape. The data may be structuredso that the first instance of each file contains the file data and isstored earlier on the tape than redundant references of the file thatreference the first instance (e.g., through a stub or pointer to thefirst instance). This allows any redundant instances of the file to beread after the actual instance of the file using a sequential readoperation. The redundant instances may contain information needed tolocate the first instance such as the offset on the current tape, or theidentifier of another tape if the backup data set spans multiple tapes.

In some embodiments, the storage system uses a similar technique whenrestoring data. For example, the storage system may restore data firstto an intermediate, random access media, such as a hard drive, where thedata can be recreated in its original form, including resolvingreferences to redundant instances of the data, before being copied to asubject system. Alternatively, the storage system may maintain the datain its single instanced form and handle requests for the data bydeinstancing the data upon request. For example, the system may copy thefirst instance of the file to a first directory, and then when a seconddirectory references the file the system copies the file from the firstdirectory based on information stored in a pointer or stub describingthe contents of second directory. These and other techniques for storingsingle instance data to sequential media are described further belowwith reference to the figures.

Figures

The details of the storage system described above will now beillustrated with reference to the figures. Unless described otherwisebelow, aspects of the invention may be practiced with conventional dataprocessing systems. Thus, the construction and operation of the variousblocks shown in FIG. 1 may be of conventional design, and need not bedescribed in further detail herein to make and use the invention,because such blocks will be understood by those skilled in the relevantart. One skilled in the relevant art can readily make any modificationsnecessary to the blocks in FIG. 1 (or other embodiments or figures)based on the detailed description provided herein.

FIG. 1 is a block diagram that illustrates components of the system, inone embodiment. The storage system 100 contains a file identificationcomponent 110, an identifier generation component 120, an identifiercomparison component 130, a single instance database component 140, arestored file cache component 150, and a storage operation managercomponent 160. The file identification component 110 identifies files orother data objects, such as in response to a storage operation. The fileidentification component 110 may retrieve additional information relatedto a data object, such as its size, used by the system to uniquelyidentify the data object. The identifier generation component 120generates a summary, or digest, of a file or other data object that isused to determine if another data object already stored by the systemmatches the data object used to generate the digest. The identifiercomparison component 130 performs comparisons of digests of various dataobjects to determine if the data objects contain similar data.

The single instance database component 140 is a data store that containsentries identifying data objects managed by the storage system 100, andmay also contain supplemental information such as a digest, path,location, reference count, or file size. The restored file cachecomponent 150 optionally provides an intermediate location used by thesystem 100 during a restore operation to hold instances of files forwhich additional references may need to be restored. For example, duringa restore operation, the storage system may restore files to the cacheand then transfer the files to a subject location of the restoreoperation. When the storage system encounters a reference to a singleinstance copy of a data object, the system may consult the restored filecache component 150 or an index to determine if the data object ispresent in the cache before attempting to restore the data object fromanother location, such as a tape. The storage operation managercomponent 160 coordinates storage operations and invokes the othercomponents of the storage system 100 as needed to perform requestedstorage operations. For example, the storage manager component 160 mayinclude an application used by an administrator to manage the system.The storage manager component 160 may also maintain indexes of the dataobjects and each of the references to those data objects through thesystem, as well as pending operations on the data objects that are partof an organization's data management plan.

FIG. 1 and the discussion herein provides a brief, general descriptionof a suitable computing environment in which the invention can beimplemented. Although not required, aspects of the invention aredescribed in the general context of computer-executable instructions,such as routines executed by a general-purpose computer, e.g., a servercomputer, wireless device or personal computer. Those skilled in therelevant art will appreciate that the invention can be practiced withother communications, data processing, or computer systemconfigurations, including: Internet appliances, hand-held devices(including personal digital assistants (PDAs)), wearable computers, allmanner of cellular or mobile phones, multi-processor systems,microprocessor-based or programmable consumer electronics, set-topboxes, network PCs, mini-computers, mainframe computers, and the like.Indeed, the terms “computer,” “host,” and “host computer” are generallyused interchangeably herein, and refer to any of the above devices andsystems, as well as any data processor.

Aspects of the invention can be embodied in a special purpose computeror data processor that is specifically programmed, configured, orconstructed to perform one or more of the computer-executableinstructions explained in detail herein. Aspects of the invention canalso be practiced in distributed computing environments where tasks ormodules are performed by remote processing devices, which are linkedthrough a communications network, such as a Local Area Network (LAN),Wide Area Network (WAN), or the Internet. In a distributed computingenvironment, program modules may be located in both local and remotememory storage devices.

Aspects of the invention may be stored or distributed oncomputer-readable media, including magnetically or optically readablecomputer discs, hard-wired or preprogrammed chips (e.g., EEPROMsemiconductor chips), nanotechnology memory, biological memory, or otherdata storage media. Indeed, computer implemented instructions, datastructures, screen displays, and other data under aspects of theinvention may be distributed over the Internet or over other networks(including wireless networks), on a propagated signal on a propagationmedium (e.g., an electromagnetic wave(s), a sound wave, etc.) over aperiod of time, or they may be provided on any analog or digital network(packet switched, circuit switched, or other scheme). Those skilled inthe relevant art will recognize that portions of the invention mayreside on a server computer, while corresponding portions reside on aclient computer such as a mobile or portable device, and thus, whilecertain hardware platforms are described herein, aspects of theinvention are equally applicable to nodes on a network.

FIG. 2 is a block diagram that illustrates the flow of data during astorage operation, in one embodiment. Data is initially stored on aserver 210 or other source of data. A storage policy 220 or otherconfiguration information specifies a storage operation to be performedon the data. For example, the storage policy 220 may specify that thedata stored on the server 210 is to be backed up daily to tape. Thestorage policy 220 causes the backup to occur, which creates the datacopy 230. The data copy 230 may contain many redundant files or otherdata objects. A media agent 240 manages the data copy 230, and creates asingle instance copy 250. The single instance copy 250 is a copy ofsource data in which the storage system has removed at least some of theredundant data objects. The media agent 240 uses the methods describedherein to eliminate redundant instances of files contained in the datacopy 230, and to produce the single instance copy 250. The singleinstance copy 250 may then be stored on tape or other media.

A media agent 240 is generally a software module that conducts data, asdirected by the storage operation manager 160, between a client computerand one or more storage devices such as a tape library, a magnetic mediastorage device, an optical media storage device, or other storagedevice. The media agent 240 is communicatively coupled with and controlsthe storage device. For example, the media agent 240 might instruct thestorage device to use a robotic arm or other means to load or eject amedia cartridge, and to archive, migrate, or restore applicationspecific data. The media agent 240 generally communicates with thestorage device via a local bus such as a SCSI adaptor. In someembodiments, the storage device is communicatively coupled to the mediaagent 240 via a Storage Area Network (“SAN”).

Each media agent 240 maintains an index cache which stores index datathe system generates during backup, migration, and restore storageoperations. For example, storage operations for Microsoft Exchange datagenerate index data. Index data provides the system with an efficientmechanism for locating user files for recovery operations. This indexdata is generally stored with the data backed up to the storage device,and the media agent 240 that controls the storage operation also writesan additional copy of the index data to its index cache. The data in themedia agent 240 index cache is thus readily available to the system foruse in storage operations and other activities without having to befirst retrieved from the storage device.

A storage policy 220 is generally a data structure or other informationwhich includes a set of preferences and other storage criteria forperforming a storage operation. The preferences and storage criteria mayinclude, but are not limited to: a storage location, relationshipsbetween system components, network pathway to utilize, retentionpolicies, data characteristics, compression or encryption requirements,preferred system components to utilize in a storage operation, and othercriteria relating to a storage operation. A storage policy 220 may bestored to a media agent index, to archive media as metadata for use inrestore operations or other storage operations, or to other locations orcomponents of the system.

FIG. 3 is a block diagram that shows a typical arrangement of thestorage system in an enterprise, in one embodiment. A typical enterprisecontains many computer systems, such as computers 300 and 310. Acomputer system 300 contains a common file 302 that is common to systemsthroughout the organization. For example common file 302 may be anoperating system file that all of the computer systems running aparticular operating system contain. The computer system 300 alsocontains a unique file A 304 that is unique to computer system 300.Similarly, computer system 310 contains another instance of the commonfile 312 and a unique file B 314 that is only stored on computer system310. A media agent 320 manages storage operations in the enterprisedescribed by storage policies, such as nightly backups of data stored oncomputer systems 300 and 310. The media agent 320 stores data in asingle instance database 330 such that common files are only storedonce. For example, the single instance database 330 contains a commonfile copy 332 that contains the same data as the instances of the commonfile 302 and 312 stored by computer systems 300 and 310. The singleinstance database 330 also contains a copy of unique file A 334 andunique file B 336.

FIGS. 4-6 and 10 are representative flow diagrams that depict processesused in some embodiments. These flow diagrams do not show all functionsor exchanges of data, but instead they provide an understanding ofcommands and data exchanged under the system. Those skilled in therelevant art will recognize that some functions or exchange of commandsand data may be repeated, varied, omitted, or supplemented, and other(less important) aspects not shown may be readily implemented.

FIG. 4 is a flow diagram that illustrates the processing of the storagemanager component 160 of the system in one embodiment. The component isinvoked when a storage operation that creates a copy of a file isrequested. In step 410, the component identifies a file to be copied,such as based on a request to backup data. In step 420, the componentdetermines whether the file is unique, or if the file has been copiedbefore. For example, the component may compute a digest in the mannerdescribed herein, and compare the digest to the digests of previouslycopied files to determine if the file is an instance of a previouslycopied file. In decision step 430, if the is unique, then the componentcontinues at step 450, else the component continues at step 440. In step440, the component adds a reference (e.g., in an index of data managedby the system) to the already backed up instance of the file, and thencompletes. In step 450, the component stores the unique file. After step450, these steps conclude.

FIG. 5 is a flow diagram that illustrates the processing of the system100 to determine whether a file is unique, in one embodiment. Thesesteps may be invoked by the storage operation component 160, such aswhen performing a storage operation as described in FIG. 4 or at othertimes. In step 510, the identifier generation component 120 generates adigest of a file that is the subject of the backup operation. In step520, the file identification component 110 gathers (e.g., by queryingthe file system of a client) additional information about the file, suchas the file's size, security information, or other attributes. In step530, the identifier comparison component 130 (e.g., on the client or amedia agent) determines if the digest of the file and any supplementalinformation matches that of any existing file tracked by the singleinstance database component 140. In decision step 540, if the filesmatch, then the system 100 continues at step 550, otherwise the system100 continues at step 570. In step 550, the system 100 reports to theentity that invoked the component that the file is not unique. In step560, the system updates the digest reference count tracked by the singleinstance database component 140 and then concludes. In step 570, thesystem 100 reports to the entity that invoked the component that thefile is unique. In step 580, the system 100 adds the file's digest andother information to the list of files tracked by the single instancedatabase component 140. These steps then conclude.

FIG. 6 is a flow diagram that illustrates the processing of the storageoperation component 160 to restore data, in one embodiment. Thecomponent is invoked when a request is received, for example, to restoredata. In step 610, the component receives a request to restore data. Instep 620, the component selects the next data object referred to by therequest. For example, the request may identify 10 data objects, and thecomponent selects the first data object on which to perform thefollowing steps. Alternatively or additionally, the 10 data objects mayhave been written to tape in various orders, and the component maygenerate a plan for accessing the 10 data objects based on adetermination of the layout of the data objects on the media. Indecision step 630, if the selected data object is a reference to aninstance of a file stored somewhere else, then the component continuesat step 640, else the component continues at step 650. In step 640, thecomponent locates the referenced instance of the file and continues tostep 655. In step 655, the component restores the data object from thereferenced instance of the file. In step 650, the component restores thefile directly from the data object. In decision step 660, if there aremore data objects referred to by the received request, then thecomponent loops to block 620 to select the next data object on which toperform these steps, else the component completes.

FIG. 7 illustrates the process of storing data from three clients firstto single-instanced random access media and then to sequential media. Insome embodiments, the system first single instances data torandom-access media, such as a magnetic disk, and then copies the singleinstanced data to sequential media, such as a tape. The system maycreate each file in a data set as one or more chunks of data, forexample, stored as one on more folders on random-access media. Client 1710 stores two files, File 1 and File 2. Client 2 720 stores two files,File 2 and File 3. Client 3 730 stores two files, File 2 and File 3. Thefiles from each client are stored in a separate chunk on the randomaccess media 740, which creates a separate folder for each chunk. Withinthe folder for the first chunk is the data for File 1 and File 2 storedby Client 1 710. This is the first time that either File 1 or File 2 hasbeen stored on the random access media 740. When client 2 stores File 2and File 3, the media agent checks its index to determine that File 2already exists in chunk 1. The media agent directs the data storageoperation to store an actual copy of File 3 in chunk 2, because File 3is being stored for the first time, and a stub or dummy file for File 2in chunk 2 that contains a pointer to the actual data for File 2 storedin chunk 1. Thus, the entire data of File 2 is only stored in Chunk 1.Chunk 1 may also contain metadata associated with File 2 that includes areference count. Each pointer to the file then increments the referencecount. The reference count may be stored with each chunk or in theindex. This allows the system to determine when particular files can areno longer referenced and can be pruned from the random access media toreclaim space. Each file or data object may have header informationstored in a tag header that stores any reference count or pointers toother objects, as well as supplemental information such as the date thefile was first stored, an expiration date after which the file can bedeleted and other storage-related information. The header informationcan be used to rebuild the index if the index is ever lost.

When client 3 stores its copies of File 2 and File 3, the media agentdetermines that both files have already been stored, and two stub filesare stored in the Chunk 3 folder. The first stub represents File 2 andpoints back to the data in Chunk 1. The second stub represents File 3and points back to the data in Chunk 2. At some point, an administratoror scheduled process determines that the data from the random accessmedia should be stored to sequential media. For example, this processcould take place as part of a nightly data storage routine. The systemstores the data for each of the chunks to the sequential media, oneafter the other. Optionally, the system may store index information onthe sequential media that identifies which data (e.g., files) is storedin which chunk. The chunks may be large and thus span multiplesequential media, and the index may contain information about thesequential media on which it is stored as well as other sequential mediathat contain data for the data set.

Alternatively or additionally, the index data may remain at the mediaagent and be protected separately as part of a data protection processused by the media agent. It should be noted that the index data is notneeded to restore data from the chunks stored on the sequential media.To the contrary, the stub information stored within each chunk issufficient to recreate the index data from sequential media, and canserve as an additional copy for protecting the index data.

FIGS. 8-9 illustrate some of the data structures used by the system.While the term “field” and “record” are used herein, any type of datastructure can be employed. For example, relevant data can have precedingheaders, or other overhead data preceding (or following) the relevantdata. Alternatively, relevant data can avoid the use of any overheaddata, such as headers, and simply be recognized by a certain byte orseries of bytes within a serial data stream. Any number of datastructures and types can be employed herein.

FIG. 8A is a data structure that illustrates the layout of storage dataon a tape or other media, in one embodiment. The data structure 800contains a series of entries that are smaller data structures, describedin more detail in FIG. 8B. The data structure 800 contains headerinformation 810 that identifies the data that follows and the media. Forexample, the header information 810 may store a serial number or otheridentifier that uniquely identifies the tape so that other tapes (ordatabase or index entries) can reference data stored on the tape, and anindex of the data stored on the tape. The index may contain informationabout the original file system structure where the data stored on thetape was located. The entry 820 signals the start of data from adirectory X. Data following the entry 820 is placed in directory X whenit is restored. The entry 830 contains information about a file A. Theentry may contain header information describing details about the data,such as the size of the data, followed by the data itself. The entry 840similarly contains information about a file B. The entry 850 signals thestart of data from a directory Y. The entry 860 contains informationabout file C. The entry 870 is the second occurrence of file A, which isstored in both directory X and directory Y. The entry 870 stores areference to the data of file A which is stored in entry 830, ratherthan a second copy of the data.

FIG. 8B is a data structure that illustrates the type of data stored ineach file and directory entry of FIG. 8A, in one embodiment. The entrycontains a header 880 and data 890. The header 880 contains a type 882and a size 884. The type 882 indicates the type of entry and what kindof information is stored in the data 890, such as directory information,data, or a reference to data. The size 884 is just one type ofadditional information that may be contained in the header 880. Forexample, the header 880 may contain access control information, timeswhen the data was last accessed or modified, and so on. The header 880may contain additional data useful for making determinations about thedata 890 that follows the header 880. If the type 882 indicates that theentry contains file data, then the data 890 contains the actual filedata 892. If the type 882 indicates that the entry is a reference todata, then the data 890 contains reference data 894, such as an offsetto the actual file data in an earlier location of the media or othermedia (e.g., in a multiple tape backup set).

FIG. 9 is a table that illustrates the information stored by the singleinstance database component 140, in one embodiment. The table 900contains a hash column 910, a first location column 920, a referencecount column 930, a size column 940, and a time column 950. The hashcolumn 910 contains a digest value computed on a file tracked by thesystem, such as an MD5 hash. The first location column 920 contains anaddress that can be used to access the file data where it is actuallylocated. For example, the address may be a network path or a URL thatspecifies a location of a file. The reference count column 930 indicateshow many different instances or references to the data have beenreported to the single instance database component 140. Alternatively oradditionally, the reference count column 930 may contain a list oflocations where the different instances are located. The size column 940and time column 950 specify additional data that may be used inconjunction with the hash value 910 to uniquely identify a file or otherdata object, such as the file's size or last modified time. The tablecontains a first entry 960 with a hexadecimal hash value, a location of“\\server\c$\io.sys” that is a network path specifying a commonoperating system file, a reference count of 56, a size of 20,254 bytes,and a time of Jan. 1, 2000. The table also contains a second entry 970with a hexadecimal hash value, a location of “\thr\employee.pdf” thatmay be an employee handbook, a reference count of 22, a size of1,568,214 bytes, and a time of Sep. 20, 2006.

FIG. 10 is a flow diagram that illustrates the processing of the storageoperation component 160 to restore data from sequential media, in oneembodiment. In step 1010, the component receives a request to restoredata. For example, the request may indicate a single file or a computerwhose data is to be restored from a tape backup. In step 1020, thecomponent generates a list of files to be restored. In step 1030, thecomponent resolves the location of each file. For example, if the mediais tape, then the location may be a tape identifier within a set oftapes and an offset within a particular tape to the file. In the examplegiven above, this step may identify the chunk of data associated withthe computer whose data is to be restored. The chunk may contain one ormore stub files rather than the actual data requested, and an additionalstep may be performed (not shown) to resolve any pointers within thechunk to other chunks that contain the actual data. In step 1040, thelocations are sorted based on their storage location on the sequentialmedia for sequential access. For example, data that is at the beginningof the tape will be retrieved before data at the end of the tape.Sequential access provides superior access times for sequential media.In step 1050, the component restores each file from the sequentialmedia.

In some embodiments, the process described in FIG. 10 can operate in twodifferent modes depending on whether index data is available in additionto the data stored on sequential media. For example, the media agentthat originally stored the data to the sequential media may be availablewith index information about the stored data. In the first mode, theindex data is used to lay out a plan for accessing the data stored onthe sequentially media as described in step 1040. Thus, the index dataprovides an optimization whereby the actual data to be restored can beaccessed directly without retrieving stub data if data is readilyavailable elsewhere in the system. In the second mode, the system hasonly the information about what files or other data are stored in eachchunk, determines which chunks to restore, and copies the data from eachchunk to a random access media file cache. Then, the system accesseseach chunk of data to determine whether the chunks contain stub data.For any stubs, the system returns to the sequential media to retrievethe actual data referenced by the stub. This method is less efficient interms of the additional seeking of the sequential media, but requiresless external infrastructure for restoring the requested data. Each modemay be preferred in different situations. For example, the first modemay be preferred when data is being restored in its original environmentwhere the media agent and other infrastructure are available for makingthe restore more efficient, but the second mode may be preferred whendata is being restored in a new environment that does not contain all ofthe infrastructure available when the data was originally stored.

As described above, in some embodiments restored data is cached onrandom access media, and the system may check the cache before restoringany particular item from sequential media. This allows the system tosave accesses to sequential media which may be slower than random accessmedia or cause more seeking of the sequential media based on the currentposition of the sequential media. The lifetime of sequential media maydepend on the number of seeks that occur, so saving these accesses byfirst accessing the cache can have reliability benefits in addition toany speed benefits.

Because the format of data stored on sequential media is type andhardware component independent, the system can single instance data setsacross heterogeneous storage media. For example, the system can singleinstance data across different storage media (tapes, disks, and so on)or file systems (Windows, UNIX, and so on). The system can then createarchive copies of data without data redundancies using heterogeneousmedia. Additionally, the system can then restore and provide data tousers across heterogeneous systems, because the system does not dependon the applications or file systems that created the data. For example,data originally created in a UNIX environment may be stored as anarchive file that is independent of typical UNIX data types. Yearslater, the system may receive a request to recover this data from aWindows-based device. Being data type independent, the systems is ableto retrieve the file (in the archive file format), and recreate the fileas a Windows based file for recovery within the Windows environment.Similarly, the system can also recover files created by differentenvironment versions (such as recovering a Windows 95 file for a Window2003 system).

CONCLUSION

From the foregoing, it will be appreciated that specific embodiments ofthe storage system have been described herein for purposes ofillustration, but that various modifications may be made withoutdeviating from the spirit and scope of the invention. For example,although backup operations have been described, the system may be usedto reduce many types of redundant storage operations. As one example,the storage system may be employed by an Internet proxy server to reducedownloading of redundant files over the Internet by tracking a digest ofeach downloaded file and the location of a downloaded instance of thefile behind the proxy server such that subsequent requests for the filecan be serviced from the previously downloaded instance withoutaccessing the file over the Internet. Similarly, the storage systemcould be used by a file system to reduce storage space by storing asingle copy of data placed in multiple locations throughout the filesystem. Accordingly, the invention is not limited except as by theappended claims.

Unless the context clearly requires otherwise, throughout thedescription and the claims, the words “comprise,” “comprising,” and thelike are to be construed in an inclusive sense, as opposed to anexclusive or exhaustive sense; that is to say, in the sense of“including, but not limited to.” The word “coupled”, as generally usedherein, refers to two or more elements that may be either directlyconnected, or connected by way of one or more intermediate elements.Additionally, the words “herein,” “above,” “below,” and words of similarimport, when used in this application, shall refer to this applicationas a whole and not to any particular portions of this application. Wherethe context permits, words in the above Detailed Description using thesingular or plural number may also include the plural or singular numberrespectively. The word “or” in reference to a list of two or more items,that word covers all of the following interpretations of the word: anyof the items in the list, all of the items in the list, and anycombination of the items in the list.

The above detailed description of embodiments of the invention is notintended to be exhaustive or to limit the invention to the precise formdisclosed above. While specific embodiments of, and examples for, theinvention are described above for illustrative purposes, variousequivalent modifications are possible within the scope of the invention,as those skilled in the relevant art will recognize. For example, whileprocesses or blocks are presented in a given order, alternativeembodiments may perform routines having steps, or employ systems havingblocks, in a different order, and some processes or blocks may bedeleted, moved, added, subdivided, combined, and/or modified. Each ofthese processes or blocks may be implemented in a variety of differentways. Also, while processes or blocks are at times shown as beingperformed in series, these processes or blocks may instead be performedin parallel, or may be performed at different times.

The teachings of the invention provided herein can be applied to othersystems, not necessarily the system described above. The elements andacts of the various embodiments described above can be combined toprovide further embodiments.

These and other changes can be made to the invention in light of theabove Detailed Description. While the above description details certainembodiments of the invention and describes the best mode contemplated,no matter how detailed the above appears in text, the invention can bepracticed in many ways. Details of the system may vary considerably inimplementation details, while still being encompassed by the inventiondisclosed herein. As noted above, particular terminology used whendescribing certain features or aspects of the invention should not betaken to imply that the terminology is being redefined herein to berestricted to any specific characteristics, features, or aspects of theinvention with which that terminology is associated. In general, theterms used in the following claims should not be construed to limit theinvention to the specific embodiments disclosed in the specification,unless the above Detailed Description section explicitly defines suchterms. Accordingly, the actual scope of the invention encompasses notonly the disclosed embodiments, but also all equivalent ways ofpracticing or implementing the invention under the claims.

While certain aspects of the invention are presented below in certainclaim forms, the inventors contemplate the various aspects of theinvention in any number of claim forms. For example, while only oneaspect of the invention is recited as embodied in a computer-readablemedium, other aspects may likewise be embodied in a computer-readablemedium. Accordingly, the inventors reserve the right to add additionalclaims after filing the application to pursue such additional claimforms for other aspects of the invention.

1-20. (canceled)
 21. A method in a computer system for restoring datafrom a deduplicated copy, the method comprising: receiving a request torestore to a destination location one or more data objects originallylocated within a hierarchy, wherein the copy of data containsinformation describing the hierarchy, but only contains a reducedinstance of each data object of the hierarchy for storage on sequentialmedia, and wherein the request specifies an original location within thehierarchy of the data objects to be restored; for each data object inthe request: determining whether the data at the original locationwithin the hierarchy specified by the request contains the data objectitself or a reference to a data object and the data object is storedelsewhere; wherein the request contains a digest value for identifyingeach data object, wherein the determining comprises comparing the digestvalue to an index of digest values of data objects stored within thecomputer system; if the original location within the hierarchy specifiedby the request contains a reference to a data object, resolving thereference to find the data object and copying the data object to thedestination location; and if the original location within the hierarchyspecified by the request contains the data object itself, copying thedata object to the destination location.
 22. The method of claim 21wherein the hierarchy is a file system hierarchy, and wherein the copyof data contains a reduced instance of the data of each file in the filesystem hierarchy, and wherein additional instances of each file in theoriginal location are stored as stubs within the copy of data thatreference the single instance of the data of each file.
 23. The methodof claim 21 wherein the reference further comprises a pointer to thelocation of the actual data for the data object, and wherein resolvingthe reference comprises locating the actual data based on the pointer.24. The method of claim 21 wherein the one or more data objects areemail messages and wherein the hierarchy comprises folders within anemail message store.
 25. The method of claim 21 wherein the copy of datais stored on one or more sequential media, wherein the reference furthercomprises a media identifier and offset of the actual data object, andwherein resolving the reference comprises locating the media identifiedby the media identifier and seeking to the offset.
 26. The method ofclaim 21 wherein the copying the data object to the destination locationfurther comprises decrypting the data object.
 27. A system for improvingthe retrieval of a data object in a network environment that containsmultiple instances of the data object, the system comprising: at leastone processor; at least one memory, coupled to the processor; means,coupled among the at least one processor and memory, for receiving arequest to access the data object at a first location; means, coupledamong the at least one processor and memory, for determining at leastone alternative location of an instance of the requested data object,wherein the at least one alternative location is not the first location;means, coupled among the at least one processor and memory, forselecting an instance of the data object from among the determined atleast one alternative location to use to satisfy the request; and means,coupled among the at least one processor and memory, for sending theselected instance of the data object in response to the request,wherein, coupled among the at least one processor and memory, therequest contains a digest value for identifying the data object, andwherein, coupled among the at least one processor and memory,determining the at least one alternative location of an instance of therequested data object comprises comparing the digest value to an indexof digest values of data objects stored within the network environment.28. The system of claim 27 wherein a computer system from which therequest is received is closer to the selected alternative location thanto the first location.
 29. The system of claim 27 wherein the request isan HTTP request for a data object located on the world wide web, andwherein the alternative location is a location within a private localarea network.
 30. The system of claim 27 wherein the means for selectingan instance of the data object comprises means for determining thefastest location from which the data object is available.
 31. The systemof claim 27 wherein the means for selecting an instance of the dataobject comprises means for identifying multiple copies containinginstances of the data object and determining attributes of each copythat affect an accessing speed for the data object.
 32. The system ofclaim 27 wherein the means for sending the selected instance of the dataobject comprises means for copying the data object from the selectedalternative location to an intermediate location, and then from theintermediate location to a location specified in the request.
 33. Amethod for controlling a computer system to recover data, wherein thecomputer system includes a processor and memory, by a method comprising:retrieving a backup object identifying a data object, wherein the backupobject includes at least one sequential data storage medium, and,wherein the at least one sequential data storage medium includes a datastructure containing information identifying whether the backup objectcontains either the data object or contains a reference to the dataobject, and; determining whether the backup object contains either thereference to the data object or contains the data object itself based onthe data structure of the at least one sequential data storage medium;when the backup object refers to the data object stored in a locationoutside of the at least one sequential data storage medium, then—locating the data object at the location outside of the at least onesequential data storage medium and copying the data object torandom-access data storage media; and when the backup object containsthe data object, copying the data object from the at least onesequential data storage medium to the random-access data storage media,wherein a first instance of each data object is stored as a copy of thedata object and each additional instance is stored as a reference to thecopy of the data object, wherein the reference is stored on the at leastone sequential data storage medium in the data structure, and, whereinthe reference refers to the location on the at least one sequential datastorage medium where the copy of the data object is stored.
 34. Thecomputer-readable medium of claim 33 wherein copying the data object tothe random-access data storage media comprises first copying the dataobject to an intermediate restored file cache, wherein the restored filecache contains a copy of the data object with any references to otherdata objects resolved.
 35. The computer-readable medium of claim 33wherein retrieving a backup object comprises retrieving a magnetic tape,and wherein the data structure contains information for building anindex of each of the copies of the data objects and references toreferences to copies of the data objects maintained by the computersystem.
 36. A non-transitory computer-readable medium containinginstructions for controlling a computer system for storing data objectsto sequential media, wherein the computer system includes a processorand memory, comprising: receiving a set of data objects, wherein the setof data objects includes at least two similar data objects; for at leastsome of the data objects in the set— determining if a copy of a dataobject is already stored on at least one random-access media, whereinthe determining includes accessing an index that contains, for dataobjects already stored on the random-access media— an identifier of thedata object; and a location of the data object on the random-accessmedia, and, when the data object is already stored on the random-accessmedia, then— identifying a location of the data object on therandom-access media and storing on the random-access media a referenceto the identified location; and, when the data object is not alreadystored on the random-access media, then storing the data object on therandom-access media; receiving a request to copy the data objects andthe references stored on the random-access media to sequential media;and copying to the sequential media, by the computer system, the dataobjects and reference data related to the references from therandom-access media, wherein the reference data on the sequential mediarefer to locations of the data objects on the sequential media.
 37. Thenon-transitory computer-readable medium of claim 36 wherein thedetermining comprises: computing a hash value of the data object; andcomparing the computed hash value to another hash value of a data objectpreviously stored by the system.
 38. The non-transitorycomputer-readable medium of claim 36 wherein the location associatedwith the sequential media is a magnetic tape that is part of a tapelibrary that includes the destination location.
 39. The non-transitorycomputer-readable medium of claim 36 wherein storing a referencecomprises storing index information containing at least a hash value ofthe data and a reference count indicating a number of references to thedata.
 40. The non-transitory computer-readable medium of claim 36wherein the reference comprises an address that points to a locationdifferent from a source location where the data object is alreadystored.
 41. The non-transitory computer-readable medium of claim 36wherein a source location comprises a disk of a client computer systemand a destination location for the copy comprises a tape libraryassociated with a storage server.
 42. The non-transitorycomputer-readable medium of claim 36 wherein a source location is afirst non-production copy of the data object and a destination locationfor the copy is a secondary non-production copy of the data object. 43.The non-transitory computer-readable medium of claim 36 wherein a sourcelocation is backup data containing multiple instances of the dataobject.